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DETAILED ACTION 
Applicant(s) Response to Official Action 

[1] The response filed on July 21, 2008 has been entered and made of record. 

Response to Arguments/Amendments 

[2] Presented arguments have been fully considered, but are rendered moot in view of the new 
ground(s) of rejection necessitated by amendment(s) initiated by the applicant(s). 

Claim Rejections - 35 USC §101 

35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any 
new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of 
this title. 

[3] Claims 20-38 are rejected under 35 U.S.C. 101 as not falling within one of the four statutory 
categories of invention. While the claims recite a series of steps or acts to be performed, a statutory 
"process" under 35 U.S.C. 101 must (1) be tied to another statutory category (such as a particular 
apparatus), or (2) transform underlying subject matter (such as an article or material) to a different 
state or thing (Reference the May 15, 2008 memorandum issued by Deputy Commissioner for 
Patent Examining Policy, John J. Love, titled "Clarification of 'Processes' under 35 U.S.C. 101"). 
The instant claims neither transform underlying subject matter nor positively tie to another statutory 
category that accomplishes the claimed method steps, and therefore do not qualify as a statutory 
process. 
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If the applicants want a more detailed explanation please contact the Office of Patent Legal 
Administration (OPLA) at (571) 272-7701. Since, Examiner is required follow this memorandum 
and cannot elaborate further than what it recites. 

Claim Rejections - 35 USC §103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 

rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 
102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the 
subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary 
skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the 

Note: US 6,956,569 is used as a translation for JP 2001-283222, hence the rejection is based on the 
Japanese application. 

[4] Claims 1, 3, 4, 19, 20, 22, 23, 38, 39, 41, 42 and 57 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Roy et al. (US 6,956,569) in combination with Nagao et al. (US 6,638,811). 

Regarding claims 1, 20 and 39, Roy discloses [a]n image matching system for retrieving a 
reference image similar to an input image, the image matching method comprising: 

means for making a first match between the input image and a plurality of representative 
three-dimensional object models ("matching a 2D image to one of a plurality of 3D candidate 
models contained in a database in which an object is identified", Roy, column 3, line 46); 

means for making a second match between the reference image and the plurality of the 
representative three-dimensional object models ("successively rendering each three dimensional 
candidate model in the determined position and orientation using the surface normals in 
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conjunction with the corresponding computed representation of lighting effects; and comparing the 
two dimensional image with each of the rendered three dimensional candidate models", Roy, 
column 4, line 6). 

Roy does not explicitly teach a means for retrieving the reference image similar to the input 
image based on the first match and the second match. 

Nagao, working in the same field of endeavor of face recognition, does teach a step of 
retrieving the reference image similar to die input image by using a result of the match between the 
input image and the representative three-dimensional object models and a result of the match 
between the reference image and the representative three- dimensional object models ("a display 17 
for displaying a face image determined in the computer system 14 as a recognized result or a collated 
result; and an output terminal 18 for outputting the face image determined in the computer system 
14", Nagao, column 14, line 19). 

It would have been obvious at the time the invention was made, for one of ordinary skill in 
the art to combine the reference image display of Nagao with the face recognition system of Roy to 
permit an operator to further verify that the system has made a valid match. 

Regarding claims 3, 22 and 41, the combination of Roy and Nagao teaches a means for 
determining a reference three-dimensional object model associated with the reference image similar 
to the input image ("matching a 2D image to one of a plurality of 3D candidate models contained in 
a database in which an object is identified", Roy, column 3, line 46); 

a conversion means for equating an input condition of the input image with an input 
condition of the reference image by converting the input image and/ or the reference image based on 
the determined reference three-dimensional object model ("successively rendering each three 
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dimensional candidate model in the determined position and orientation using the surface normals 
in conjunction with the corresponding computed representation of lighting effects", Roy, column 4, 
line 6); and 

means for retrieving the reference image similar to the input image by making a third match 
between the input image and the reference image equated to the input condition of the input image 
("comparing the two dimensional image with each of the rendered three dimensional candidate 
models", Roy, column 4, line 10) . 

Regarding claims 19, 38 and 57, the combination of Roy and Nagao teaches [fjhe image 
matching system according to claim 1, wherein an object of the plurality of the representative three- 
dimensional object models is a is a human face ("Although this invention is applicable to numerous 
and various types of objects to be recognized, it has been found particularly useful in the 
environment of human face recognition", Roy, column 5, line 15). 

Claims 4, 23 and 42, the combination of Roy and Nagao discloses substantially the claimed 
invention as set forth in the discussion above for claim 22. 

The combination of Roy and Nagao does not disclose expressly converting the reference 
image prior to the means for making the second match making the second match. 

At the time of the invention, it would have been obvious to a person of ordinary skill in the 
art to convert the input image. Applicant has not disclosed that converting the input image rather 
than the reference image provides an advantage, is used for a particular purpose or solves a stated 
problem. One of ordinary skill in the art, furthermore, would have expected Applicant's invention 
to perform equally well with either converting the reference image or the input image. 
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Therefore, it would have been obvious to combine to one of ordinary skill in this art to 
modify the combination of Roy and Nagao to obtain the invention as specified in claims 4, 23 and 
42. 

[5] Claims 2, 5, 9, 14, 28, 24, 21, 33, 40, 43, 47 and 52 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Roy et al. (US 6,956,569) and Nagao et al. (US 6,638,811) in combination 
with Savakis et al. (US 6,847,733). 

Regarding claims 2, 21 and 40 the combination of Roy and Nagao teaches the image 
matching method according to claim 20. 

The combination of Roy and Nagao does not teach further comprising: 

means for determining a reference three-dimensional object model associated with the 
reference image similar to the input image; and 

means for retrieving an updated reference image similar to the input image by using the 
determined reference three-dimensional object model and the input image. 

Savakis, working in a similar problem solving area of retrieving images from a database, does 
teach a step of finding a reference three-dimensional object model associated with the reference 
image similar to the input image; and 

a step of newly retrieving the reference image similar to the input image by using the 
reference three-dimensional object model and the input image ("... provide selected images of 
highest emphasis or appeal 1228 hack to the query processor 1222 to perform another database 
search", Savakis, column 15, line 64 and figure 13, Savakis teaches the concept of using feedback to 
improve upon an image match within a database). 
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It would have been obvious at the time the invention was made for one of ordinary skill in 
the art to combine the pattern matching feedback element of Savakis in the face recognition system 
of the combination of Roy and Nagao to find a better match by providing candidate reference 
images similar to the initial match ("The feedback feature 1228 can be used to provide a broader 
search/retrieval of the database 1200 by utilizing the highest emphasis and appeal images 1228 as 
the query instead of the single query image 1220", Savakis, column 1 5, line 66). 

Regarding claims 5, 24 and 43, the combination of Roy, Nagao and Savakis teaches [tjhe 
image matching system according to claim 1, comprising: an image input means for inputting the 
input image ("a query photograph 50 is presented to the system for identification", Roy, column 10, 
line 34); representative three-dimensional object model storage section for storing the plurality of 
the representative three-dimensional object models ("Asynchronously, and over arbitrary lengths of 
time, the system is presented with 3D objects 10, typically human heads and preferably human faces, 
which it scans and digitizes with a scanning device 20, at the same time capturing the reflected light 
image at all included 3D data points. A reflectance estimation module 30 computes a reflectance 
function for the 3D object 10. This information is then stored in a database 40", Roy, column 10, 
line 23); first image generation means for generating at least one comparison image close in input 
condition to the input image for each representative three-dimensional object model among the 
plurality of the representative three-dimensional object models based on the plurality of the 
representative three-dimensional object models stored in the representative three- dimensional 
object model storage section ("it obtains the next model 50 from the database 40, and obtains from 
a pose module 120 the pose solution for that model 10 and query image 50", Roy, column 10, line 
45); 
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first image matching means for calculating similarities between the input image and the at 
least one comparison image of the each representative three-dimensional object model generated by 
the image generation means and selecting the at least one comparison image of a representative 
three- dimensional object model among the plurality of the representative three dimensional object 
models, which has a greatest similarity with the input image based on the similarities calculated by 
the first image matching means ("The lightsphere module 60 has a sub module 140 which 
sequentially considers one model 10 at a time for similarity to the query image 50", Roy, column 10, 
line 43); 

a reference image storage section for storing reference images of objects ("... provide 
selected images of highest emphasis or appeal 1228 back to the query processor 1222 to perform 
another database search", Savakis, column 15, line 64 and figure 13, in order for Savakis to provide a 
reference image back to the query processor, the images must have been previously stored, figure 13, 
reference 1200); 

a step of storing similarities between the reference images stored in the reference image 
storage section and representative three-dimensional object models stored in the representative 
three-dimensional object model storage section, in a reference image matching result storage section 
("The lightsphere module 60 has a sub module 140 which sequentially considers one model 10 at a 
time for similarity to the query image 50", Roy, column 10, line 43, in order to keep track of an 
earlier best match, there must be some provision for retaining the results); and 

a result matching means for extracting reference images similar to the input image based on 
similarities between the input image and the at least one comparison image of the each 
representative three-dimensional object model calculated by the first image matching means and the 
similarities between the reference images and the plurality of the representative three-dimensional 
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object models stored in the reference image matching result storage section ("a display 17 for 
displaying a face image determined in the computer system 14 as a recognized result or a collated 
result; and an output terminal 1 8 for outputting the face image determined in the computer system 
14", Nagao, column 14, line 19). 

Regarding claims 9, 28 and 47, the combination of Roy, Nagao and Savakis teaches [t]he 
image matching method according to claim 21, comprising: 

an image input step of inputting die input image ("a query photograph 50 is presented to the 
system for identification", Roy, column 10, line 34); 

a step of storing the plurality of the representative three-dimensional object models in a 
representative three-dimensional object model storage section ("Asynchronously, and over arbitrary 
lengths of time, the system is presented with 3D objects 10, typically human heads and preferably 
human faces, which it scans and digitizes with a scanning device 20, at the same time capturing the 
reflected light image at all included 3D data points. A reflectance estimation module 30 computes a 
reflectance function for the 3D object 10. This information is then stored in a database 40", Roy, 
column 10, line 23); 

a first image generation step of generating at least one comparison image close in input 
condition to the input image for each representative three-dimensional object model among the 
plurality of the representative three-dimensional object models based on the plurality of the 
representative three-dimensional object models stored in the representative three- dimensional 
object model storage section ("it obtains the next model 50 from the database 40, and obtains from 
a pose module 120 the pose solution for that model 10 and query image 50", Roy, column 10, line 
45); 
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a first image matching step of calculating similarities between the input image and the at least 
one comparison image of the each representative three-dimensional object model generated by the 
first image generation step, and selecting the at least one comparison image of a representative three- 
dimensional object model among the plurality of the representative three-dimensional object 
models, which has the greatest similarity as a similarity with the input image based on the similarities 
calculated by the first image matching step ("The lightsphere module 60 has a sub module 140 
which sequentially considers one model 10 at a time for similarity to the query image 50", Roy, 
column 10, line 43); 

a step of storing the reference images of objects in a reference image storage 

section ("... provide selected images of highest emphasis or appeal 1228 back to the query 
processor 1222 to perform another database search", Savakis, column 15, line 64 and figure 13, in 
order for Savakis to provide a reference image back to the query processor, the images must have 
been previously stored, figure 13, reference 1200); 

a step of storing similarities between the reference images stored in the reference image 
storage section and the plurality of the representative three-dimensional object models stored in the 
representative three-dimensional object model storage section, in a reference image matching result 
storage section ("The lightsphere module 60 has a sub module 140 which sequentially considers one 
model 10 at a time for similarity to the query image 50", Roy, column 10, line 43, in order to keep 
track of an earlier best match, there must be some provision for retaining the results); 

a result matching step of extracting reference images similar to the input image based on the 
similarities between the input image and the at least one comparison image of the each 
representative three-dimensional object model calculated by the first image matching step and the 
similarities between the reference images and the plurality of the representative three-dimensional 
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object models stored in the reference image matching result storage section ("a display 17 for 
displaying a face image determined in the computer system 14 as a recognized result or a collated 
result; and an output terminal 1 8 for outputting the face image determined in the computer system 
14", Nagao, column 14, line 19); 

a step of storing reference three-dimensional object models associated with each reference 
image among the reference images stored in the reference image storage section, in a reference 
three-dimensional object model storage section ("Asynchronously, and over arbitrary lengths of 
time, the system is presented with 3D objects 10, typically human heads and preferably human faces, 
which it scans and digitizes with a scanning device 20, at the same time capturing the reflected light 
image at all included 3D data points. A reflectance estimation module 30 computes a reflectance 
function for the 3D object 10. This information is then stored in a database 40", Roy, column 10, 
line 23); 

a second image generation step of obtaining reference three-dimensional object models 
associated with reference images extracted at the result matching step, from the reference three- 
dimensional object model storage section, and generating at least one second comparison image 
close in input condition to the input image every reference three-dimensional object model on the 
basis of the obtained reference three-dimensional object models ("... provide selected images of 
highest emphasis or appeal 1228 back to the query processor 1222 to perform another database 
search", Savakis, column 15, line 64 and figure 13, in order for Savakis to provide a reference image 
back to the query processor, the images must have been previously stored, figure 13, reference 
1200); and 

a second image matching step of calculating similarities between the input image and second 
comparison images generated at the second image generation step, selecting a maximum similarity 
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from among second comparison images associated with each of the reference three-dimensional 
object models, and regarding the maximum similarity as a similarity between the input image and the 
reference three-dimensional object model ("The lightsphere module 60 has a sub module 140 which 
sequentially considers one model 10 at a time for similarity to the query image 50", Roy, column 10, 
line 43). 

Regarding claims 14, 33 and 52, the combination of Roy, Nagao and Savakis teaches [t]he 
image matching method according to claim 22, comprising: 

an image input step of inputting the input image ("a query photograph 50 is presented to the 
system for identification", Roy, column 10, line 34); 

a step of storing a plurality of representative three-dimensional object models in a 
representative three-dimensional object model storage section ("Asynchronously, and over arbitrary 
lengths of time, the system is presented with 3D objects 1 0, typically human heads and preferably 
human faces, which it scans and digitizes with a scanning device 20, at the same time capturing the 
reflected light image at all included 3D data points. A reflectance estimation module 30 computes a 
reflectance function for the 3D object 10. This information is then stored in a database 40", Roy, 
column 10, line 23); 

an image generation step of generating at least one comparison image close in input 
condition to the input image every representative three-dimensional object model on the basis of the 
representative three-dimensional object models stored in the representative three- dimensional 
object model storage section ("it obtains the next model 50 from the database 40, and obtains from 
a pose module 120 the pose solution for that model 10 and query image 50", Roy, column 10, line 
45); 
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an image matching step of calculating a similarity between the input image and each of the 
comparison images generated by the image generation means, selecting a maximum similarity with 
respect to comparison images associated with each representative three- dimensional object model, 
and regarding the maximum similarity as a similarity between the input image and the representative 
three-dimensional object model ("The lightsphere module 60 has a sub module 140 which 
sequentially considers one model 10 at a time for similarity to the query image 50", Roy, column 10, 
line 43); 

a step of storing the reference images of objects in a reference image storage 

section ("... provide selected images of highest emphasis or appeal 1228 back to the query 
processor 1222 to perform another database search", Savakis, column 15, line 64 and figure 13, in 
order for Savakis to provide a reference image back to the query processor, the images must have 
been previously stored, figure 13, reference 1200); 

a step of storing similarities between the reference images stored in the reference image 
storage section and representative three-dimensional object models stored in the representative 
three-dimensional object model storage section, in a reference image matching result storage section 
("The lightsphere module 60 has a sub module 140 which sequentially considers one model 10 at a 
time for similarity to the query image 50", Roy, column 10, line 43, in order to keep track of an 
earlier best match, there must be some provision for retaining the results); 

a result matching step of extracting the reference image similar to the input image on the 
basis of similarities between the input image and the representative three-dimensional object models 
calculated by the image matching means and similarities between the reference images and the 
representative three-dimensional object models stored in the reference image matching result storage 
section ("a display 17 for displaying a face image determined in the computer system 14 as a 



Application/Control Number: 10/520,661 Page 14 

Art Unit: 2624 

recognized result or a collated result; and an output terminal 1 8 for outputting the face image 
determined in the computer system 14", Nagao, column 14, line 19) 

a step of storing reference three-dimensional object models associated with the reference 
images stored in the reference image storage section, in a reference three-dimensional object model 
storage section ("Asynchronously, and over arbitrary lengths of time, the system is presented with 
3D objects 10, typically human heads and preferably human faces, which it scans and digitizes with a 
scanning device 20, at the same time capturing the reflected light image at all included 3D data 
points. A reflectance estimation module 30 computes a reflectance function for the 3D object 10. 
This information is then stored in a database 40", Roy, column 10, line 23); 

an image conversion step of obtaining reference three-dimensional object models associated 
with reference images extracted at the result matching step, from the reference three- dimensional 
object model storage section, squaring an input condition of the input image with that of the 
reference image extracted at the result matching step by converting the input image and/ or the 
reference image extracted at the result matching step, on the basis of the obtained reference three- 
dimensional object models, and generating partial images respectively of the input image and the 
reference image squared in input condition with each other ("determining the position and 
orientation of an object giving rise to the two dimensional image; computing a representation of 
lighting effects that allows the lighting that gave rise to the two dimensional image to be used to 
render a realistic image of a three dimensional model; successively rendering each three dimensional 
candidate model in the determined position and orientation using the surface normals in 
conjunction with the corresponding computed representation of lighting effects; and comparing the 
two dimensional image with each of the rendered three dimensional candidate models", Roy, 
column 4, line 2); and 
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a partial image matching step of calculating a similarity between the partial image of the 
input image and the partial image of the reference image generated at the image conversion step 
("comparing the two dimensional image with each of the rendered three dimensional candidate 
models", Roy, column 4, linelO). 

Allowable Subject Matter 

[6] Claims 6-8, 10-13, 15-18, 25-27, 29-32, 34-37, 44-46, 48-51 and 53-56 are objected to as 
being dependent upon a rejected base claim, but would be allowable if rewritten in independent 
form including all of die limitations of die base claim and any intervening claims and above 101 
rejections are resolved. 

Contact Information 

[7] Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Mr. Sath V. Perungavoor whose telephone number is (571) 272.-7455. The 
examiner can normally be reached on Monday to Friday from 8:30am to 5:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Mr. Matthew C. Bella whose telephone number is (571) 272-7778, can be reached on Monday to 
Friday from 9:00am to 5:00pm. The fax phone number for the organization where this application 
or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR system, 
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see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, 
contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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